REPORT  DOCUMENTATION  PAGE 


Form  Approved 
0MB  No.  0704-0188 


Public  reporting  burden  for  this  collection  of  information  is  estimated  to  average  1  hour  per  response,  Including  the  time  for  reviewing  instructions,  searching  existing  data 
sources,  gathering  and  maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  only,  other 
aspect  of  this  collection  of  information,  Including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and 
Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington,  VA  22202-4302,  and  to  the  Office  of  Management  and  Budget,  Paperwork  Reduction  Project  (07804-0188), 
Washington,  DC  20503.  _ — — 


1 .  AGENCY  USE  ONLY  (LEAVE  BLANK)  2.  REPORT  DATE  3.  REPORT  TYPE  AND  DATES  COVERED 


4,  TITLE  AND  SUBTITLE 

Flight  Test  Automation  Options 


Professional  Paper 


5.  FUNDING  NUMBERS 


6.  AUTHOR(S) 

Dean  Carico 


7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS{ES) 

Naval  Air  Warfare  Center  Aircraft  Division 
22347  Cedar  Point  Road,  Unit  #6 
Patuxent  River,  Maryland  20670-1 161 


9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 


8.  PERFORMING  ORGANIZATION 
REPORT  NUMBER 


10.  SPONSORING/MONITORING 
AGENCY  REPORT  NUMBER 


11.  SUPPLEMENTARY  NOTES 

12a.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  is  unlimited. 

12b.  DISTRIBUTION  CODE 

13.  ABSTRACT  (Maximum  200  words) 

14.  SUBJECT  TERMS 

15.  NUMBER  OF  PAGES 

16.  PRICE  CODE 

17.  SECURITY  CLASSIFICATION  18.  SECURITY  CLASSIFICATION  19.  SECURITY  CLASSIFICATION  20.  LIMITATION  OF  ABSTRACT 
OF  REPORT  OF  THIS  PAGE  OF  ABSTRACT 


Unclassified 


Unclassified 


Unclassified 


SAR 


Flight  Test  Automation  Options 


CLEARED  fOR 
OPEN  PUBLICATION 

MAR  1  9  1998 


Dean  Carico 

Test  and  Evaluation  Engineering  4.1 1.3.2 
Naval  Air  Warfare  Center  Aircraft  Division 
Patuxent  River  MD  20670  ii- 
Phone;  (301)  342-1382 


PUBUC  AFFAIRS  OFFICE 
NAVAL  AIR  SYSTEMS  COMMAND 


Abstract 

Flight  testing  is  often  called  the  key  component  of  test  and  evaluation.  The  cost  of  conventional 
flight  testing  is  expected  to  escalate  approaching  the  2T'  century  and  beyond.  Augustine  noted 
several  years  ago  that  if  this  trend  continues,  a  single  advanced  fighter  aircraft  would  cost  more 
than  the  entire  DoD  budget  by  the  middle  of  next  century.  As  the  cost  of  flight  testing  continues 
to  escalate  in  a  predicted  hostile  fiscal  environment,  it  is  important  to  consider  options  to  help 
minimize  flight  test  cost.  Suggestions  range  fi'om  completely  eliminating  developmental  testing 
to  employing  a  variety  of  flight  test  automation  options.  Flight  test  automation  option  concepts 
range  from  the  fantasy  of  “push  a  button,  the  test  is  done,”  to  the  more  practical  use  of  a 
personal  computer  to  help  with  some  repetitive  flight  test  tasks  and  to  help  store  large  amounts  of 
related  data.  Options  to  help  automate  specific  aspects  of  flight  testing  are  starting  to  gain 
acceptance.  Several  test  automation  options  exist  that  have  the  potential  to  enhance  flight  testing 
by  permitting  it  to  be  done  better,  faster,  cheaper,  and  safer.  This  paper  briefly  discusses  a 
variety  of  flight  test  automation  options  including  the  OSD  Automated  Test  Planning  System 
(ATPS)  work  to  automate  the  test  and  evaluation  master  plan  (TEMP),  the  Army  Test  and 
Evaluation  Planning  and  Reporting  System  (TEPRS),  the  G&C  System  work  on  Test_Plan,  the 
Calspan/Air  Force  Test  Planning,  Analysis,  and  Evaluation  System  (PAES)  program,  the  Boeing 
Planning  and  Reporting  Organizer  For  Test  Engineers  (PROFITE)  program,  and  the  Naval  Air 
Warfare  Center  Aircraft  Division  Automated  Flight  Test  Engineering  System  (AFTES)  program 
and  high  performance  computing  program  on  flight  test  automation. 

Background 

Powered  aircraft  flight  testing  has  been  ongoing  for  almost  a  century,  but  new  lessons  still  appear 
to  be  learned  for  each  new  flight  test  program.  At  the  start  of  the  20*  century,  aircraft 
entrepreneurs  were  jacks  of  all  trades,  who  designed  the  aircraft,  built  the  aircraft  and  performed 
the  flight  testing.  At  this  stage,  the  flight  test  procedures  and  flight  test  requirements  were  being 
developed.  As  the  aircraft  became  more  complex,  specialized  flight  test  crews  were  required, 
along  with  specialized  test  and  data  analysis  techniques. 

In  1943,  many  Navy  activities  were  consolidated  at  Patuxent  River,  Md.,  to  form  the  Naval  Air 
Test  Center  (NATC)  (currently  the  Naval  Air  Warfare  Center  Aircraft  Division  (NAWCAD)). 
Initial  work  toward  standardizing  test  techniques  and  test  requirements  began  with  the 
establishment  of  the  US  Navy  Test  Pilot  School  (USNTPS),  which  supported  NATC  test 
directorates.  Navy  rotorcraft  testing  in  the  late  1960’s  was  performed  using  techmques  defined 
in  the  Rotary  Wing  Branch  Test  Manual  (1959)  and  USNTPS  test  manuals.  Data  reduction,  at 
this  time,  was  accomplished  using  a  adding  machine  plus  a  desktop  calculator  which  could  be 
programmed  with  up  to  15  steps.  This  was  a  big  step  forward  for  new  engineers  who  had  just  got 
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through  college  using  slide  rules.  Hand  calculators  and  desk  top  calculators  were  used  for  data 
reduction  during  the  1980’s.  A  preliminary  test  report  and  test  plan  database  was  developed 
during  this  time  frame  by  the  Rotary  Wing  Sea  Control  Department.  Personal  computers  showed 
up  at  the  Rotary  Wing  Aircraft  Test  Directorate  in  the  1980’s,  and  during  this  timeframe  a 
survey  indicated  that  it  could  take  up  to  a  month  to  develop  a  comprehensive  rotorcraft  flying 
qualities  and  performance  test  plan.  With  the  personal  computer,  it  was  possible  to  take  an 
approved  test  plan  and  change  the  helicopter  using  a  search  and  replace  function  of  the  word 
processor  software.  Data  reduction  techniques  could  be  programmed  in  general  terms  and  could 
be  used  for  different  test  programs.  New  engineers  and  pilots  were  continually  entering  the 
system,  and  developing  methodologies  to  capture  the  knowledge  of  the  “test  experts”  before 
they  retired  or  changed  jobs  was  considered  important.  Thus,  flight  test  automation  options  have 
been  evolving  since  the  turn  of  the  20*  century.  If  the  DoD  defense  budget  had  continued  to 
increase  after  the  cold  war  with  more  new  aircraft  being  tested  and  procured  each  year,  there 
would  have  been  little  incentive  to  radically  change  the  way  flight  testing  had  been  traditionally 
performed. 

Introduction 

Current  factors  such  as  declining  budgets,  reduced  staffs,  increased  project  cost,  and  tightened 
delivery  schedules  all  point  to  the  need  to  improve  the  current  flight  test  process.  The  cost 
associated  with  the  next  generation  rotorcraft  testing,  training,  and  support,  using  current 
techniques,  promises  to  escalate  in  a  predicted  hostile  fiscal  environment.  Vision  21  [1]  calls  for 
a  reduction  in  the  current  test  and  evaluation  (T&E)  infrastructure.  The  Simulation,  Test  and 
Evaluation  Process  (STEP)  [2]  and  DoD  Regulation  5000.2-R  [3]  require  modeling  and 
simulation  throughout  the  system  life  cycle.  Zittel  [4]  reviews  the  DoD  Simulation  Support  Plan 
which  calls  for  “...  increasing  emphasis  on  the  use  of  modeling  and  simulation  (M&S)  in  our 
acquisition  programs  to  reduce  cost  and  schedule  without  sacrificing  quality  or  performance. 
Simulation  based  acquisition  is  considered  an  effective,  affordable  mechamsm  for  fielding 
complex  technologies  and  may  help  to  make  DoD  a  “smart  buyer  [5].  Programs  associated  with 
some  form  of  flight  test  automation  have  been  ongoing  for  some  time. 

Automation  Systems 

Automated  Test  Planning  System  (ATPS)  -  This  program  was  sponsored  by  the  Deputy 
Director,  Systems  Engineering  and  Evaluation  (Test  and  Evaluation)  in  the  Office  of  Under 
Secretary  of  Defense  for  Acquisition  and  Technology  at  OSD.  The  program  focus  includes 
using  expert  system  based  tools  to  assist  in  reviewing  or  drafting  a  Test  and  Evaluation  Master 
Plan  (TEMP)  and  in  assessing  Test  &  Evaluation  (T&E)  program  risk.  This  SAIC  program  [6] 
called  Automated  Test  Planning  System  (ATPS)  involves  a  joint  approach  to  enhance  and 
standardize  TEMP  development  and  review.  ATPS  includes  the  following  modules: 

TEMP  Build  Module 

TEMP  Review  Module 

Test  and  Evaluation  Program  Risk  Assessment  Module 

Test  and  Evaluation  Program  Design  Module 


The  ATPS  program  contains  the  DoD  5000  series  documents  to  support  T&E  and  acquisition 
programs,  as  shown  in  figure  1.  The  software  is  available  for  both  PC  and  Macintosh  computers 
and  can  be  downloaded  from  the  World  Wide  Web.  The  program  requires  approximately  4  MB 
of  storage  and  8  MB  of  RAM.  The  ATPS  program  has  over  600  registered  users  supporting 
multi-service  programs.  The  ATPS  has  been  used  successfully  on  the  F-22  program. 

Test  and  Evaluation  Planning  and  Reporting  System  (TEPRS)  -  This  program,  originally 
known  as  the  Test  and  Evaluation  Plan  (TEP)  Builder,  was  developed  by  the  Army  Test  and 
Evaluation  Command,  PRC  Inc.,  and  the  University  of  Michigan  to  support  Army  operational 
test  and  evaluation  (OT&E).  The  TEP  Builder  includes  several  different  programs,  including  the 
knowledge  base  and  tool  set.  The  TEP  was  designed  primarily  for  new  testers  with  strong 
service-related  expertise,  but  little  familiarity  with  the  OT&E  process  [7].  The  program  is 
currently  designated  TEPRS,  and  focuses  on  T&E  planning  and  reporting.  A  functional  block 
diagram  of  TEPRS  is  presented  in  figure  2.  The  TEPRS  primary  functions  include: 

Create  definitions  for  systems  test  and  evaluation 
Manage  definition  consistency  across  all  test  documents 
Provide  commercial-of-the-self  (COTS)  software  integration 

The  TERPS  program  is  designed  to  operate  on  a  personal  computer  running  Windows  software. 
The  program  requires  6MB  of  hard  drive  space  for  installation,  and  includes  a  run  time  version 
of  Microsoft  Access.  TERPS  is  undergoing  Beta  testing  and  future  versions  will  include  such 
features  as  spreadsheet  test  matrix  design  options,  presentation  options,  a  repository  of  test 
factors,  system  evaluation  report  type  documents,  test  design  and  analysis  options,  and  data 
element  dictionaries. 

TEST_PLAN  -  This  software  package  for  workstation  computers  is  developed  by  G&C 
Systems,  Inc.,  to  help  flight  test  engineers  (FTE)  plan  and  track  flight  test  programs  by  mapping 
requirements  to  test  points,  flight  test  maneuvers,  and  flights  [8].  TEST_PLAN  is  designed  to 
reduce  the  cost  of  flight  testing  by  maximizing  the  productive  use  of  available  flight  time, 
minimising  planning  and  scheduling  errors,  providing  management  visibility  into  program  status. 

TEST_PLAN  was  initially  demonstrated  in  1989,  and  has  been  commercially  available  for  five 
years.  It  was  used  at  Saab  Scania  on  the  JAS-39  and  Saab  2000  flight  test  programs,  and  at  the 
NASA  Dryden  Flight  Research  Facility,  Edwards  Air  Force  Base  (AFB)  on  F-18  research 
projects.  In  addition,  the  Air  Force  Flight  Test  Center  (AFFTC),  Edwards  AFB,  the  C-17 
Combined  Test  Force  and  the  C-130  Combined  Test  Force  at  AFFTC  are  using  TEST_PLAN. 
The  Boeing/Lockheed/General  Dynamics  F-22  Team  has  also  purchased  TEST_PLAN  for  the  F- 
22  flight  test  program.  Hindustan  Aeronautics  Limited  is  using  TEST_PLAN  on  their  Light 
Combat  Aircraft  (LCA)  flight  test  program. 

TEST_PLAN  is  a  large  software  package  with  extensive  graphics  and  must  be  hosted  on  high- 
end  graphics  workstation  computers  running  under  UNIX  or  VMS  for  satisfactory  performance. 
TEST_PLAN  is  integrated  with  ORACLE,  a  commercially  available  relational  database 
management  system.  TEST_PLAN  is  currently  being  ported  to  PC  computers  running  Windows 


NT  4.0  or  subsequent.  This  version  should  be  released  in  mid  1998.  TEST_PLAN  is  an 
evolving  system.  Current  work  is  centered  around  providing  better  support  for  avionics  system 
test  planning  and  requirements  tracking. 

A  diagram  of  the  flight  test  planning  functionality  supported  by  TEST_PLAN  is  presented  in 
Figure  3.  TEST_PLAN  automatically  generates  documentation  (flight  card  decks,  flight  reports, 
instrumentation  reports)  in  FrameMaker,  Interleaf  or  Microsoft  Word.  TEST_PLAN  contains  a  3 
DOF  simulation  facility  capable  of  generating  flight  profiles  for  a  given  airplane  and  displaying 
them  on  digital  maps.  The  system  requires  standard  aerodynamic,  propulsion  and  geometric  data 
on  the  aircraft  to  be  simulated. 

Test  Planning,  Analysis,  and  Evaluation  System  (PAES)  -  Test  PAES  is  a  software  package 
for  personal  computers  developed  by  Calspan  SRL  Corporation  for  the  US  Air  Force  Armstrong 
Laboratory  [9-12].  The  program  was  initiated  in  1992  with  initial  software  delivery  in  1993,  and 
with  field  testing  commencing  in  early  1994,  and  is  described  in  ITEA  Journal  Vol  19,  No.  1  of 
March/ April  1998.  The  software  consists  of  four  components,  as  summarized  in  figure  4,  and 
the  program  is  currently  located  at  over  50  test  sites  in  the  US. 

Planning  and  Reporting  Organizer  For  Test  Engineers  (PROFTE)  -  PROFTE  is  a  Microsoft 
Access  and  Word  based  software  package  for  personal  computers  developed  by  the  Boeing 
Aerospace  Group  (formerly  McDoiuiell  Douglas)  to  support  the  F/A-18E/F  flight  test  program 
[13].  The  criteria  for  the  software  package  included  the  ability  to  support  over  20,000  test 
points,  over  2000  flights,  seven  aircraft,  six  test  sites,  multi-users,  and  no  budget  outside  normal 
test  planning  work.  PROFTE  was  designed  to  help  the  test  team  members  prepare  test  plan 
work  descriptions,  prepare  flight  cards,  track  test  points,  prepare  flight  reports,  and  report  test 
program  progress.  Test  persoimel  from  the  F-18A-D,  AV-8B,  F15A-E,  T-45,YF23,  and  B-2 
programs  provided  lessons  learned  and  suggestions  for  PROFTE  design  and  development. 
PROFTE  has  been  used  over  the  past  three  years  to  support  over  1600  F/A-18E/F  flights. 

The  primary  PROFTE  databases  include  a  test  point  tracking  system,  flight  card/reporting 
database,  external  loadout  database,  loadout  scheduling  database,  and  flutter  control  unit  program 
management  database,  as  shown  in  figure  5.  The  test  point  tracking  system  database  is  the  heart 
of  the  system  and  it  contains:  test  points,  test  point  status,  test  point/aircraft  assignments,  test 
work  description,  test  attempts,  test  attempt  status,  and  flight  time.  The  flight  card/reporting 
database  is  used  by  test  team  members  for  test  mission  planning  parameters  and  test  mission 
reporting  parameters.  This  database  also  provides  aircraft  metrics  and  flight  logs.  The  external 
loadout  database  provides  information  on  all  external  stores  configurations.  The  loadout 
scheduling  database  is  used  to  schedule  aircraft  loading  and  stores  delivery  and  provide  Navy 
flight  schedule  documentation.  The  flutter  exciter  control  unit  database  is  used  to  support  flutter 
and  parameter  identification  testing. 

Automated  Flight  Test  Engineering  System  (AFTES)  -  This  concept  was  initiated  as  a  low- 
key,  in-house  Navy  Patuxent  River  test  plan  development  program  in  1987  to  use  new  software 
to  support  fixed-wing  electro-optics  testing.  The  program  focus  shifted  to  rotorcraft 
helicopter/ship  and  flying  qualities  and  performance  testing  in  FY89,  and  the  concept  was  called 


the  Test  Plan  Automation  System  (TP AS).  Different  expert  shell  structures  were  evaluated  and 
limited  work  was  performed  to  input  flight  test  knowledge  base  information  into  the  TPAS 
system.  Funding  for  this  initial  work  ended  in  1992,  and  in  1994  a  small  business  innovative 
research  (SBIR)  program  was  initiated  that  renewed  the  effort.  Stottler,  Henke,  and  Associates, 
Inc.  (SHAI),  is  prime  on  the  SBIR  and  it  focuses  on  using  artificial  intelligence  technology  to 
enhance  rotorcraft  test  planning,  test  reporting,  data  analysis,  and  project  management.  This 
SBIR  program  project  is  called  the  Automated  Flight  Test  Engineering  System  (AFTES).  The 
AFTES  concept  is  illustrated  in  figure  6,  and  should  be  available  for  use  during  FY98.  Test  team 
members  can  select  up  to  nine  different  types  of  helicopter/ship  or  dynamic  interface  tests  with  a 
beta  version  of  AFTES,  and  after  answering  a  few  questions,  generate  a  comprehensive  test  plan 
in  approximately  15  minutes  using  a  Pentium  PC.  A  second  SBIR  program  focuses  on 
developing  a  PC  tutorial  to  support  rotorcraft  handling  qualities,  flight  control  system,  and  loads 
testing.  Integrating  the  AFTES  program  with  a  variety  of  flight  test  analysis  tools,  as 
summarized  in  figure  7,  forms  one  option  for  a  rotorcraft  flight  test  automation  program. 

High  Performance  Computing  Flight  Test  Automation  Program  -  The  purpose  of  this 
program  is  to  develop  advanced  simulation  software  that  can  be  interfaced  with  high 
performance  computing  hardware  to  help  automate  aircraft/systems  testing.  This  program 
provides  a  unified  environment  to  support  current  and  next  generation  aircraft/systems  testing. 
The  work  involves  integrating  and  enhancing  advanced  technology  programs  in  simulation 
modeling  and  flight  test  automation  to  form  a  unified  environment  to  enhance  testing.  Current 
rotorcraft  simulation  models  are  vehicle  type  specific  and  do  not  have  the  high  fidelity  rotor  and 
fuselage  components  required  to  predict  loads.  The  current  modeling  environment  also  does  not 
support  helicopter/ship  operational  envelope  development  due  to  limitations  in  current  aircraft 
modeling  and  ship  environment  modeling.  A  physics-based  generic  structure  simulation  model, 
with  selectable  levels  of  fidelity,  has  the  potential  to  support  testing  on  a  wide  variety  of  multi¬ 
service  rotorcraft.  Elastic  rotor  blade  and  fuselage  components,  combined  with  complex  ship 
airwake  models,  require  high  performance  computing  (HPC)  hardware  for  potential  real-time 
operations,  as  well  as  code  parallelization,  and  code  optimization  using  HPC  hardware.  Aircraft 
model  code  enhancements  are  required  to  better  support  flight  testing  and  to  take  advantage  of 
available  HPC  hardware.  Code  enhancements  are  required  to  better  support  basic  flying 
qualities  and  performance  testing,  helicopter/ship  dynamic  interface  testing,  and  reliability  and 
maintainability  testing.  A  variety  of  artificial  intelligence  (AI)  technologies  could  be  combined  to 
help  automate  test  planning,  data  analysis,  and  test  reporting.  The  test  plan  contains  a  flight  test 
matrix  appendix  which  lists  all  the  planned  flights  along  with  flight  test  conditions  and  data  plot 
formats.  Integrating  the  test  plan  automation  program  with  a  high  fidelity  simulation  model  on  a 
high  performance  computing  system  would  allow  the  test  flights  to  be  run  analytically  while  the 
test  plan  was  being  generated.  The  HPC  system  could  be  used  to  support  real-time  test  team 
training  or  numerous  parameter  sweeps  at  each  data  point  using  the  complex  aircraft  model. 
This  parameter  sweep  information  could  be  used  to  provide  quick  answers  to  sponsor  questions 
on  “What  happens  if  I  change  ..x..  or  ..y..  on  my  aircraft?”  The  ability  to  do  comprehensive 
flight  test  programs  and  design  trade-off  studies  analytically  has  the  potential  of  large  savings  for 
the  test  community.  The  capability  has  broad  multi-service  and  commercial  test  applications, 
with  initial  focus  on  Army,  Navy,  and  Air  Force  rotorcraft/systems  testing.  The  generic  nature 


of  the  proposed  software  make  it  portable,  scaleable,  and  reusable.  It  also  satisfies  the  high 
priority  DoD  need  to  do  testing  better,  faster,  and  cheaper,  and  to  help  reduce  the  overall  test  and 
evaluation  infrastructure. 

Summary 

Flight  testing  is  and  will  continue  to  remain  a  very  interesting  and  challenging  occupation  for 
future  generations.  The  test  team  knowledge  base  required  for  safe  and  effective  flight  testing 
continues  to  increase  with  increasing  aircraft  complexity.  Declining  budgets  and  test  support 
personnel  emphasize  the  importance  of  options  to  improve  test  program  efficiency  while 
retaining  safe  test  procedures.  Several  flight  test  automation  options  exist,  in  various  stages  of 
development,  testing,  and  application,  which  are  designed  to  help  test  team  members  do  more 
with  limited  resources  available.  These  test  automation  programs  have  the  potential  to  permit 
future  flight  testing  to  be  done  better,  faster,  cheaper,  and  safer. 
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Figure  1  Automated  Test  Planning  System  (ATPS)  Overview 


Figure  2  Test  and  Evaluation  Planning  and  Reporting  System  (TEPRS)  Overview 
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Figure  3  TEST_PLAN  Flight  Test  Planning  Functionality 


Figure  4  Test  Planning,  Analysis,  and  Evaluation  System  (PAES)  Components 


A.  Prepare  Test  Work  Description(TWD)  -  Test  Plan 

B.  Prepare  Flight  Cards 


C.  Report  Results 

D.  Report  Progress 


Figure  5  Planning  and  Reporting  Organizer  For  Test  Engineers  (PROFTE)  Overview 
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Figure  6  Automated  Flight  Test  Engineering  System  (AFTES)  Overview 
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Figure  7  Rotorcraft  Flight  Test  Automation  Summary 


